# Human-Robot Commensality Dataset

## Overview
A novel audio-visual dataset capturing social eating behaviors of groups of three people sharing a meal while a robot feeds one of them. The dataset contains multi-view RGBD video and directional audio recordings of 10 sessions comprising 30 people. We provide the data in a ROS bag format.

Please see Section 7 (and the referenced Appendix sections) of [our paper](https://arxiv.org/abs/2207.03348) [1] for details on the user study setup, procedure, and questionnaires.

## Naming conventions
The 10 recorded sessions are denoted by session IDs from 01 to 10. The session ID 00 corresponds to the pilot study. 

## Documentation pictures
Pictures capturing the study setup details can be found in the folder `documentation-pictures`. 

## ROS bag files and associated extraction scripts
The ROS bag files are provided only for participants who agreed their data will be made publicly available.
Due to size limitations, we provide the ROS bag files externally at https://cornell.box.com/v/human-robot-commensality-bag<br>
For each session there is one ROS bag file with the filename `{session_ID}.bag` and it contains the following ROS topics:
 - /audio - mixed audio
 - /audio/channel0 - mixed audio
 - /audio/channel1 - mic 1 audio
 - /audio/channel2 - mic 2 audio
 - /audio/channel3 - mic 3 audio
 - /audio/channel4 - mic 4 audio    
 - /camera1/aligned_depth_to_color/image_raw/compressed  
 - /camera1/color/image_raw/compressed                   
 - /camera2/aligned_depth_to_color/image_raw/compressed  
 - /camera2/color/image_raw/compressed                   
 - /camera3/aligned_depth_to_color/image_raw/compressed  
 - /camera3/color/image_raw/compressed                   
 - /sound_direction - in degrees (see the documentation pictures)

The microphone and camera positions follow the setup in Human-Human Commensality Dataset, except we did not use a scene camera in this study.

The scripts to extract video and audio data based on the specified ROS topics can be found in the folder `bag-extraction-scripts`.

## Bite timing conditions
The bite timing conditions for each of the 10 sessions and 10 trials are recorded in the file `conditions.csv`. The three types of bite timing conditions are:
- a: Fixed-Interval Timing
- b: Mouth-Open Timing
- c: Learned Timing
The bite-timing conditions were also generated for two extra sessions if needed.

## Questionnaires
The data from the pre-study, experiment, and post-study questionnaires is located in the folder `questionnaires`.
The replies to open-ended post-study questions can also be found in `questionnaires/user_study_openended_quetions_replies.txt` in an easy to read format.

---
## References

[1] Ondras, Jan, Abrar Anwar, Tong Wu, Fanjun Bu, Malte Jung, Jorge Jose Ortiz, and Tapomayukh Bhattacharjee. "Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups." In 6th Annual Conference on Robot Learning. 2022. https://openreview.net/forum?id=7ZcePvChS7u

    @inproceedings{ondras2022humanrobot,
      title={Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups},
      author={Jan Ondras and Abrar Anwar and Tong Wu and Fanjun Bu and Malte Jung and Jorge Jose Ortiz and Tapomayukh Bhattacharjee},
      booktitle={6th Annual Conference on Robot Learning},
      year={2022},
      url={https://openreview.net/forum?id=7ZcePvChS7u}
    }